Access token used for Service Extension no longer accessible from REST API

#1

Hi,

we use Slack extensions on more than a hundred Pagerduty services and up to now were managing them using the method described here:
/forum/t/how-to-pass-config-data-when-creating-an-extension/851/2

We used this method through the Terraform Pagerduty provider as described here: https://github.com/terraform-providers/terraform-provider-pagerduty/issues/94#issuecomment-449075707

The issue is this does not seem to be working any more since yesterday as the token is now redacted from the REST API responses:

image

From the terraform point of view, this triggers permanent diff:

# terraform plan

  # module.pagerduty.pagerduty_extension.slack_nbh["MyService103"] will be updated in-place
  ~ resource "pagerduty_extension" "slack_nbh" {
      ~ config            = jsonencode(
          ~ {
              ~ access_token     = "redacted" -> "xoxp-******-******-******-******"
                bot              = {
                    bot_user_id = "U******1"
                }
                enterprise_id    = null
                incoming_webhook = {
                    channel           = "#fr-******-pagerduty"
                    channel_id        = "C******X"
                    configuration_url = "https://******.slack.com/services/******"
                    url               = "https://hooks.slack.com/services/******/******/******"
                }
                notify_types     = {
                    acknowledge = false
                    annotate    = true
                    assignments = false
                    escalate    = true
                    resolve     = false
                    trigger     = true
                }
                ok               = true
                referer          = "https://******.pagerduty.com/services/******/integrations"
                scope            = "identify,bot,commands,incoming-webhook,channels:read,groups:read,im:read,team:read,users:read,users:read.email,channels:write,chat:write:user,chat:write:bot,groups:write"
                team_id          = "T******Y"
                team_name        = "******"
                urgency          = {
                    high = true
                    low  = true
                }
                user_id          = "U******6"
            }
        )
        extension_objects = [
            "P******7",
        ]
        extension_schema  = "P******R"
        id                = "P******P"
        name              = "MyService103 - #fr-******-pagerduty in ******.slack.com"
    }

Plan: 0 to add, 103 to change, 0 to destroy.

Sadly, all our Slack V2 extensions that shared a previously valid access_token stopped working all at once and we cannot re-authorize them through the API.

Is there any other known method to retrieve an access token?

2 Likes
Authorize Slack Extension for a Service with Pager Duty Rest API
#2

Alright, the new method to retrieve this access token is during the OAuth re-authentication phase in an actual browser:

  1. In Firefox, browse to a service integrations page, e.g. https://******.pagerduty.com/services/P*****C/integrations,
  2. On a Slack V2 extension, open the drop down menu by clicking the gear icon,
  3. Click Re-authorize,
  4. Open the Developer Tools with F12,
  5. Execute this in the Developer Tools Console: document.body.getElementsByTagName("script")[1].text
  6. Extract the token value from the text, it should look like xoxs-************-************-************-****************************************************************
  7. Select the target Slack channel and click Allow.
2 Likes
#3

I ran into the exact same problem. What should be done with the xoxs token then?

I mean, did you find a way to use this new token to re-authorize all the extensions using the API/Terraform?

Thanks a lot for posting this. It’s very helpful.

#4

My understanding is that you should replace “access_token” with the new “xoxs-*” token inside the config section, but I couldn’t get it to work myself.

We have faced exactly the same issue on Feb 6th, I have manually re-authorised extension for one of the services while getting new token. However, when I use that token as “access_token” to propagate config via Terraform for other services it gets applied successfully, but not Slack alerts are created when incident is triggered.

Obscuring “access_token” value makes it very difficult to troubleshoot any further, so help will be much appreciated.

Thanks.

#5

Hello all,

Thank you for bringing this to our attention! If you haven’t already, could you open tickets with us in Support by emailing support@pagerduty.com and referencing this community post?

We’re happy to investigate here!

Nick - Support

#6

Please see #240748. Many thanks.

#7

When I tried on Feb 7th, it worked for some but not all, so I ended up re-authorizing all slack extensions by hand as I was leaving for a one week vacation.

Got the information something was fixed by Pagerduty engineering team during my vacation (ticket #240113).

I’m planning to perform more tests soon.

One thing to expect with terraform for now until some better solution is implemented is permanent diff as the token is redacted.

#8

Hi Patrick and all!

I want to apologize for the delay in reaching out. We have been looking into possible ways to recover the ability to authorize the Slack extension via Terraform/REST API and remove the manual authorization required through the web UI.

There is a missing aspect bot_access_token needed within the bot field seen in this Curl request below:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' --header 'Authorization: Token token=REDACTED' -d '{
    "extension": {
        "name": "NAME",
        "config": {
            "notify_types": {
                "resolve": true,
                "trigger": true,
                "escalate": true,
                "acknowledge": true,
                "assignments": true,
                "annotate": true
            },
            "urgency": {
                "high": true,
                "low": false
            },
            "referer": "https://REDACTED.pagerduty.com/extensions",
            "ok": true,
            "scope": "identify,bot,commands,incoming-webhook,channels:read,groups:read,im:read,team:read,users:read,users:read.email,channels:write,chat:write:user,chat:write:bot,groups:write,pins:write",
            "user_id": "REDACTED",
            "team_id": "REDACTED",
            "enterprise_id": null,
            "team_name": "REDACTED",
            "incoming_webhook": {
                "channel": "REDACTED",
                "channel_id": "REDACTED",
                "configuration_url": "https://REDACTED.slack.com/services/REDACTED",
                "url": "https://hooks.slack.com/services/REDACTED/REDACTED/REDACTED"
            },
            "bot": {
                "bot_user_id": "REDACTED",
                "bot_access_token": "REDACTED"
            },
            "access_token": "REDACTED"
        },
        "extension_schema": {
            "id": "PII3QUR",
            "type": "extension_schema_reference"
        },
        "extension_objects": [{
            "id": "REDACTED",
            "type": "service"
        }],
        "type": "webhook",
        "summary": "NAME"
    }
}' 'https://api.pagerduty.com/extensions'

To use this work-around, one can generate a Legacy Personal token from the Slack website.

You would use the same personal token within the bot_access_token and access_token. These two fields are not included within the REST API when you get the details of an existing Slack extension.

There are a few caveats to keep in mind when using this method:

  • OAuth is not officially supported for extension_schemas through the REST API making this solution subject to change
  • The legacy personal Slack token is tied to a user which will be removed from Slack when the user is removed from the Slack instance
  • The bot_user_id is unique to each Slack workspace so you will need to obtain this from a working Slack extension. I used the REST API in this case.

We’re hoping this work-around will help in achieving your use-case but definitely open to any feedback and suggestions.

2 Likes
#9

@geeth Thanks for revealing information about the bot_access_token config field!

Passing the same values from both user_id and access_token (starting with xoxp-) in the bot.bot_user_id bot.bot_access_token config fields does make it work again :slight_smile:

Example terraform configuration:

locals {
  extension_config_slack = {
    for team_name, team in var.teams : team_name => {
      user_id      = team.pagerduty_slack_extension["user_id"]
      access_token = team.pagerduty_slack_extension["access_token"]

      # Using the same values as above makes it work again
      bot = {
        bot_user_id      = team.pagerduty_slack_extension["user_id"]
        bot_access_token = team.pagerduty_slack_extension["access_token"]
      }

      enterprise_id = null

      incoming_webhook = {
        channel           = team.pagerduty_slack_extension["incoming_webhook_channel"]
        channel_id        = team.pagerduty_slack_extension["incoming_webhook_channel_id"]
        configuration_url = team.pagerduty_slack_extension["incoming_webhook_configuration_url"]
        url               = team.pagerduty_slack_extension["incoming_webhook_url"]
      }

      notify_types = {
        acknowledge = false
        annotate    = true
        assignments = false
        escalate    = true
        resolve     = false
        trigger     = true
      }

      ok = true

      scope = "identify,bot,commands,incoming-webhook,channels:read,groups:read,im:read,team:read,users:read,users:read.email,channels:write,chat:write:user,chat:write:bot,groups:write"

      team_name = team.pagerduty_slack_extension["team_name"]
      team_id   = team.pagerduty_slack_extension["team_id"]

      urgency = {
        high = true
        low  = true
      }
    }
  }
}

data "pagerduty_extension_schema" "slack" {
  name = "Slack V2"
}

resource "pagerduty_extension" "slack_bh" {
  for_each = { for name, customer in var.customers : name => customer if customer.nbh != "no" }

  name = "${pagerduty_team.teams[each.value.team].name} - ${each.key} HNO - ${var.teams[each.value.team].pagerduty_slack_extension["incoming_webhook_channel"]} in ${var.teams[each.value.team].pagerduty_slack_extension["workspace_name"]}"

  extension_schema  = data.pagerduty_extension_schema.slack.id
  extension_objects = [pagerduty_service.nbh[each.key].id]

  config = jsonencode(
    merge(
      local.extension_config_slack[each.value.team],
      { "referer" = format("%s/integrations", pagerduty_service.nbh[each.key].html_url) },
    )
  )
}

Note: I’m working on an evolution of the terraform provider to let it ignore changes to the redacted configuration fields.

1 Like
#10

@geeth thanks for revealing bot_access_token, this has finally worked for me.

I don’t think “access_token” needs to be set to the same personal token as I was able to set it as “redacted” (to avoid infinite diff) and had no problem creating new extension. All in all, setting bot_access_token will suffice based on my tests.

1 Like
#11

I suppose bot_access_token not being redacted is currently a mistake as @geeth said:

That will probably change too in the future.

#12

Hey Patrick and Alex,

Glad this worked out! I didn’t try it without the access_token token in my initial testing so I’m glad to hear it’s not needed.

#13

This unfortunately didn’t work for us. Setting the “bot_access_token” seems to have altered the user that posts to the Slack channel from “Pagerduty” to the owner of the token. This would be fine, but it also seems to have broken the Slack button functionality where we can acknowledge/resolve/etc from within Slack.

4 Likes
#14

Thanks for raising this, Ryan!

I see a loading cog when attempting to send a POST/PUT to PagerDuty as well.

We’ll do some internal reviews to see how this differentiates from the requests made via the manually authorized PagerDuty app and get back to you.

4 Likes